Monocular template-based 3D surface reconstruction: Convex inextensible and nonconvex isometric methods

We study the 3D reconstruction of an isometric surface from point correspondences between a template and a single input image. The template shows the surface flat and fronto-parallel. We propose three new methods. The first two use a convex relaxation of isometry to inextensibility. They are formulated as Second Order Cone Programs (SOCP). The first proposed method is point-wise (it reconstructs only the input point correspondences) while the second proposed method uses a smooth and continuous surface model, based on Free-Form Deformations (FFD). The third proposed method uses the ‘true’ nonconvex isometric constraint and the same continuous surface model. It is formulated with Nonlinear Least-Squares and can thus be solved with the efficient Levenberg-Marquardt minimization method. The proposed approaches may be combined in a single pipeline whereby one of the convex approximations is used to initialize the nonconvex method. Our contributions solve two important limitations of current state of the art: our convex methods are the first ones to handle noise in both the template and image points, and our nonconvex method is the first one to use ‘true’ isometric constraints. Our experimental results on simulated and real data show that our convex point-wise method and our nonconvex method outperform respectively current initialization and refinement methods in 3D reconstructed surface accuracy. Code release. We intend to release our code under the GPL license. Research highlights: • A convex SOCP based method with noise in template and image • The use of a continuous surface model based on an FFD • A nonconvex, statistically founded, and truly isometric reconstruction method